Transfering Data from Couchbase to the ElasticSearch (Transport-Couchbase Plugin)


If we want to transfer the data persisted in your Couchbase to ElasticSearch and use the power of inverted indexing of Elastic Search along with Couchbase. Then we can do this in easy steps.

Couchbase provides us a plugin for the Elastic search that makes your ElasticSearch node appear like a Couchbase Server node. After installation you can use the Cross-Datacenter Replication (XDCR) feature of Couchbase Server to transfer data continuously.

Installing Couchbase-Transport plugin:

Steps for installing the Couchbase transport plugin are:

  1. Go to your installation directory for Elasticsearch:
    cd elasticsearch-<version>
  2. Install the Couchbase Plug-in. Replace the version number with the appropriate one. You can check your own required version here.
    bin/plugin -install transport-couchbase -url \
    http://packages.couchbase.com.s3.amazonaws.com/releases/ \
    elastic-search-adapter/2.1.1/elasticsearch-transport-couchbase-2.1.1.zip

    After a successful installation, the plug-in installer returns:

    DONE Installed transport-couchbase
  3. Set the user name and password for the plug-in:
    echo "couchbase.password: password" >> config/elasticsearch.yml; 
    echo "couchbase.username: Administrator" >> config/elasticsearch.yml    You can add it permanently inside your elasticsearch.yml for easy access. 
  4. Install kopf, a third-party plug-in for Elasticsearch that provides a simple web-based interface you can use to interact with Elasticsearch:
    ./elasticsearch/bin/plugin install lmenezes/elasticsearch-kopf/{branch|version}
    
  5. Start Elasticsearch:
     bin/elasticsearch 

    Elasticsearch starts and runs on your machine in the background.

Updating configurations:

  1. For making index into the elastic search you have to set a default template for your index.
    This default template json can be found here. You can edit it as per your need and change the mapping of your index accordingly as per the need.

The command for making the default template is :

curl -XPUT http://localhost:9200/_template/couchbase \ -d @plugins/transport-couchbase/couchbase_template.json

You can also set it by copying the content of  couchbase_template.json and making the  template from kopf UI like this:

Screenshot from 2016-01-15 23-17-59

2. Now make the index for the bucket whose data is to be copied.
the command for making the index is:

curl -XPUT http://localhost:9200/<index-name&gt;

For more insight in the ElasticSearch you can refer here.

Making XDCR replication from Couchbase to the ES.

Now you can make an XDCR replication on the ElasticSearch server.This can be done in small and easy steps.

  1. Make connection to the ElasticSearch server from the XDCR tab in the Couchbase UI.ElasticSearch server
  2. Add an ongoing replication to the server given above.
    1. If you are using Couchbase Server 2.2 or later, click Advanced settings and change the XDCR Protocol setting to Version 1.

    Replication

  3. You can use the advance filtering option for copying a particular type of document and can easily specify a regex for the document Id that you want to copy.RegexAdvance
  4. Start the XDCR and you are good to go, and it starts getting copied inside your ES. Which will have the whole document as well as the couchbase meta-data for each and every document.
  5. For more tuning of the elastic search you can use the following settings for your elasticSearch which can be feed up into the elasticsearch.ymlsettings

Now you are good to go to see your data inside your ElasticSearch by making a Query to the ElasticSearch. For more insight in the CRUD operations on ES and Querying ES you can refer this link. 

If you have any doubts. We’d be happy to solve them. Ping us!

About shiv4nsh

Coder, Gamer, Learner..!!
This entry was posted in Architecture, Best Practices, couchbase, Devops, Elasticsearch, integration, Scala and tagged , , , . Bookmark the permalink.

4 Responses to Transfering Data from Couchbase to the ElasticSearch (Transport-Couchbase Plugin)

  1. Pingback: Implementing full text search with Couchbase and harnessing the power of Couchbase full text search (CBFT) | Knoldus

  2. is there a reason to use couchbase with ES versus simply loading the data directly as an ES index?

    • toddbryantsr says:

      Yes, updating documents in ElasticSearch is a lot simpler in CouchBase than in ElasticSearch. Additionally, it’s easy to do something bad in ElasticSearch. Having a firewall between ES and CB gives another layer of security.

  3. Pingback: Data Fan – divyanshspacetech

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s