Working With Elasticsearch In Scala


Elasticsearch is an open-source, restful, distributed, search engine built on top of apache-lucene. You can read it more on their website.

We have an application built in scala with sbt as our build tool. Now we required to have a search capability on the output created by the application. We chose to use elasticsearch for our application.

In this post, we will learn to use elasticsearch java api in Scala. The scenario will be that we will index a json to elasticsearch. Then search the json document in elasticsearch in order to validate that it got indexed.

Let’s start by installing it. Installing Elasicsearch is simple, download it from link and unzip it. To run elasticsearch descend to unzipped directory and execute:

elasticsearch -f

“-f” tells elasticsearch to run in foreground. So, we will be able to see logs here.

Here is how it appears when it runs in foreground.

To start with the coding part, begin with adding dependency of elasticsearch in the project. At the time 0.19.8 was the latest. The artifact is available on typesafe repository. Here is the snippet in build.sbt file.

name := "elasticsearchproject"

version := "0.1.0"

scalaVersion := "2.9.2"

resolvers += "Typesafe Repo" at "http://repo.typesafe.com/typesafe/releases/"

libraryDependencies += "org.elasticsearch" % "elasticsearch" % "0.19.8"

Elasticsearch is schemaless. We can index any json to it. We have a bulk json file, each line is a json. For our implementation: Application reads file line by line and index json to the elasticsearch.

Here is the bulk json. Each line is a json.

{ "id": 1, "source": "twitter", "data": "first sample data" }
{ "id": 2, "source": "twitter", "data": "second sample data" }
{ "id": 3, "source": "twitter", "data": "third sample data" }
{ "id": 4, "source": "twitter", "data": "fourth sample data" }
{ "id": 5, "source": "twitter", "data": "fifth sample data" }
{ "id": 6, "source": "twitter", "data": "sixth sample data" }

Here is the complete application.

package com.meetu.elasticsearch

import org.elasticsearch.node.NodeBuilder.nodeBuilder
import scala.io.Source

object ElasticSearchApplication extends App {
  val node = nodeBuilder().client(true).node()
  val client = node.client()

  val jsons = Source.fromFile("src/main/resources/sample.json").getLines().toList
  for (i <- 1 to jsons.length) {
    indexJson(jsons(i - 1), i.toString)
  }

  def indexJson(json: String, id: String) = {
    client.prepareIndex("esa", "activityStream", id).setSource(json).execute().actionGet()
  }

}

We create a node using nodeBuilder. We then create a client from node created. There is a method indexJson that accepts json string and an id which is like a primary key.

in the prepareIndex call on client we pass three parameters
1) name of the index: “esa” in our case
2) document type: “activityStream” in our case
3) unique id: an unique value. We can use it for searching

We then set the source as json passed in the method and we are done. We now expect that json will be indexed by elasticsearch. Let’s search it.

Elasticsearch has rest api through which we can search documents on an index. Since it is rest api we can search by a GET request. Let’s use curl utility.

curl -XGET http://localhost:9200/esa/activityStream/1

here “http://localhost:9200&#8243; is url:port of elasticsearch. “esa” is name of index, recall that in the api call in scala code we used “esa” as index. “activityStream” is document type and last one is the id.

And here is the output of the curl request.

{"_index":"esa","_type":"activityStream","_id":"1","_version":3,"exists":true, "_source" : { "id": 1, "source": "twitter", "data": "first sample data" }}

We are done with a quick-start scala project that uses elasticsearch java api in Scala. It is a simple api and in a few lines of code we can index a json from our application in elasticsearch. You can explore more on the api here.

This entry was posted in Scala and tagged , . Bookmark the permalink.

2 Responses to Working With Elasticsearch In Scala

  1. Pingback: Complexificaton: Is ElasticSearch Making a Case for a Google Search Solution? : Stephen E. Arnold @ Beyond Search

  2. sara says:

    Very helping tutorial ,
    please provide more tutorials of elastic search using scala

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s