Transfering Data from Couchbase to the ElasticSearch (Transport-Couchbase Plugin)

If we want to transfer the data persisted in your Couchbase to ElasticSearch and use the power of inverted indexing of Elastic Search along with Couchbase. Then we can do this in easy steps.

Couchbase provides us a plugin for the Elastic search that makes your ElasticSearch node appear like a Couchbase Server node. After installation you can use the Cross-Datacenter Replication (XDCR) feature of Couchbase Server to transfer data continuously.

Installing Couchbase-Transport plugin:

Steps for installing the Couchbase transport plugin are:

1. Go to your installation directory for Elasticsearch:

   cd elasticsearch-<version>

2. Install the Couchbase Plug-in. Replace the version number with the appropriate one. You can check your own required version here.

   bin/plugin -install transport-couchbase -url \
   http://packages.couchbase.com.s3.amazonaws.com/releases/ \
   elastic-search-adapter/2.1.1/elasticsearch-transport-couchbase-2.1.1.zip

   After a successful installation, the plug-in installer returns:

   DONE Installed transport-couchbase

3. Set the user name and password for the plug-in:

   echo "couchbase.password: password" >> config/elasticsearch.yml; 
   echo "couchbase.username: Administrator" >> config/elasticsearch.yml    You can add it permanently inside your elasticsearch.yml for easy access. 

4. Install kopf, a third-party plug-in for Elasticsearch that provides a simple web-based interface you can use to interact with Elasticsearch:

   ./elasticsearch/bin/plugin install lmenezes/elasticsearch-kopf/{branch|version}
   

5. Start Elasticsearch:

    bin/elasticsearch 

   Elasticsearch starts and runs on your machine in the background.

Updating configurations:

1. For making index into the elastic search you have to set a default template for your index.
   This default template json can be found here. You can edit it as per your need and change the mapping of your index accordingly as per the need.

The command for making the default template is :

curl -XPUT http://localhost:9200/_template/couchbase \ -d @plugins/transport-couchbase/couchbase_template.json

You can also set it by copying the content of  couchbase_template.json and making the  template from kopf UI like this:

2. Now make the index for the bucket whose data is to be copied.
the command for making the index is:

curl -XPUT http://localhost:9200/<index-name&gt;

For more insight in the ElasticSearch you can refer here.

Making XDCR replication from Couchbase to the ES.

Now you can make an XDCR replication on the ElasticSearch server.This can be done in small and easy steps.

1. Make connection to the ElasticSearch server from the XDCR tab in the Couchbase UI.
2. Add an ongoing replication to the server given above.
   1. If you are using Couchbase Server 2.2 or later, click Advanced settings and change the XDCR Protocol setting to Version 1.

   

3. You can use the advance filtering option for copying a particular type of document and can easily specify a regex for the document Id that you want to copy.
4. Start the XDCR and you are good to go, and it starts getting copied inside your ES. Which will have the whole document as well as the couchbase meta-data for each and every document.
5. For more tuning of the elastic search you can use the following settings for your elasticSearch which can be feed up into the elasticsearch.yml

Now you are good to go to see your data inside your ElasticSearch by making a Query to the ElasticSearch. For more insight in the CRUD operations on ES and Querying ES you can refer this link. 

If you have any doubts. We’d be happy to solve them. Ping us!