Neo4j Apoc : A Blessing For Developer

Hello Folks,

As we know about Neo4j, it pulls out developers life from the trouble and black & white screen of the databases. It doesn’t give freedom from the old databases also provides best support with it’s predefined procedures.

As we know that in the Relational Database, Procedure provide advantages of better performance, scalability, productivity, ease of use and security and Neo4j also provides some amazing tool which can perform as mention above.

Yes, I am talking about the Apoc and using of Apoc with Neo4j, is a blessing for the developers. It provides many predefined procedures or user defined functions/views so that we can easily use it and improve our productivity in very simple manner.

APOC is stands for ‘Awesome Procedure On Cypher‘. APOC is a library of procedure for the various areas. It is introduce with the Neo4j 3.0

There are many areas where we use APOC and the lit of areas are :

• Graph Algorithm
• Metadata
• Manual indexes and relationship indexes
• Full text search
• Integration with other databases like MongoDB, ElasticSearch, Cassandra and relational databases
• Path expansion
• Import and export
• date and time function
• Loading of XML and JSON from APIs and files
• String and text function
• Concurrent and batched Cypher Execution
• spatial Function and Lock
• Collection and map utilities

When you are using APOC there are two ways to get it and use with Neo4j :

First Way :

• Download binary jar from the latest release [Hit Here]
• Put that into your $Neo4j_Home/plugins/ folder
• Restart your Neo4j Server.

Second Way :

• Clone neo4j-apoc-procedure from Hit Here.
• Go to the folder with ‘cd neo4j-apoc-procedures’.
• Now create a jar with the help of command ‘mvn clean compile install’.
• Now copy your jar file from target to $Neo4j_Home/plugins/ folder.[cp target/apoc-1.0.0-SNAPSHOT.jar $Neo4j_Home/plugins/]
• Restart your Neo4j Server.

Now you are ready to use APOC with Neo4j. Today we will discuss about the data migration between the other data base to Neo4.

We use many databases for storing the data. But when we have a large amount of data and tables that time it becomes so hard to make query and execute them on the database. We have to be extra cautious to perform the task and we get bored to see same screen without any fun 🙂 . When we worked on any other database and think to use neo4j that time we face the issue for migrating data into Neo4j. We are going to discuss migrating data from some famous and use databases.

Oracle :

We are in the last database to migrate data to Neo4j but as obvious not least. We can download JDBC .jar file (Download) and keep it in the $Neo4j_Home/plugins and restart the Neo4j. We can provide URL in the $Neo4j_Home/conf/neo4j.conf as :

apoc.jdbc.oracle_url.url=jdbc:oracle:thin:user/password@127.0.0.1:1521/XE

• After restarting the Neo4j server we are set for migrating the data from the Oracle to Neo4j. We fetch the data from the Oracle where we have a table with name employee_details to Neo4j.Now we load the driver with the APOC.

CALL apoc.load.jdbc('oracle_url','employee_details') YIELD row
RETURN count(*);

• Let’s create Index, Constraints and Relation the data.

/**
* Here we define schema and key.
*/
CALL apoc.schema.assert(
 {EMPINFO:['name', 'age','salary']},
 {EMPINFO:['id'],ADDRESS:['address']});

• Now we will load data and perform Merge and Create operation so that we can create the node and relationship between the node.

/**
* Here we load data in the neo4j and create node with the help of schema which we define
* earlier.
*/
CALL apoc.load.jdbc('oracle_url','employee_details') yield row
MERGE (g:ADDRESS {name:row.ADDRESS})
CREATE (t:EMPINFO {id:toString(row.ID), name:row.NAME, age:toString(row.AGE), salary:toString(row.SALARY)})
CREATE (t)-[:LIVE]->(g);

• We can see Relation Graph and it will look something like this :

/**
* For Displaying Performed Relation
*/

MATCH p=()-[r:LIVE]->() RETURN p LIMIT 25;

MYSQL :

We want to migrate data from the MYSQL as before we have to download JDBC .jar file (Download) and keep it in the $Neo4j_Home/plugins and update $Neo4j_Home/conf/neo4j.conf as:

apoc.jdbc.mysql_url.url=jdbc:mysql://localhost:3306/test?user=user&password=pass

Restart the Neo4j server and we are set for migrating the data from the Cassandra to Neo4j.

• We hit the MySQL and start fetching data and perform count operation.

CALL apoc.load.jdbc('mysql_url','employee_data') yield row
RETURN count(*);

PostgreSQL :

When we use PostgreSQL, we have to download JDBC .jar file (Download) and keep it in the $Neo4j_Home/plugins and restart the Neo4j. After restarting the Neo4j server we are set for migrating the data from the PostgreSQL to Neo4j.

• Now we load the driver with the APOC.

CALL apoc.load.driver('org.postgresql.Driver');

• Now we create the call for fetching the data from the PostgreSQL where we have a table with name employee_details to Neo4j.

with 'jdbc:postgresql://localhost:5432/testdb?user=postgres&password=postgres' as url
CALL apoc.load.jdbc(url,'employee_details') YIELD row
RETURN count(*);

• If we don’t want to use these step than we can provide URL in the $Neo4j_Home/conf/neo4j.conf and restart the server :

apoc.jdbc.postgresql_url.url=jdbc:postgresql://localhost:5432/testdb?user=postgres&password=postgres

We can now fetch data direct. We don’t need to load driver also.

CALL apoc.load.jdbc('postgresql_url','employee_details') YIELD row
RETURN count(*);

• Create Nodes and Relation in the data.

/**
* Here we define schema and key. In first column we define those column_name
* which can be null and In the second we those column name which we want unique.
*/

CALL apoc.schema.assert( {Detail:['name','age','address','salary']},
{Detail:['id']});

/**
* Here we load data in the neo4j and create node with the help of schema which we define
* earlier.
*/

CALL apoc.load.jdbc('jdbc:postgresql://localhost:5432/testdb?user=postgres&password=postgres','employee_details') yield row
CREATE (t:Detail {id:toString(row.id), name:row.name,
age:toString(row.age), address:row.address, salary:toString(row.salary)})
return t;

Cassandra :

Now we migrate data from the Cassandra to Neo4j. Now we first import data into the cassandra if we don’t have data in the cassandra or we can use it for test also.

• We have to run following command for setting up initial data in the cassandra :

curl -OL https://raw.githubusercontent.com/neo4j-contrib/neo4j-cassandra-connector/master/db_gen/playlist.cql
curl -OL https://raw.githubusercontent.com/neo4j-contrib/neo4j-cassandra-connector/master/db_gen/artists.csv
curl -OL https://raw.githubusercontent.com/neo4j-contrib/neo4j-cassandra-connector/master/db_gen/songs.csv
$CASSANDRA_HOME/bin/cassandra
$CASSANDRA_HOME/bin/cqlsh -f playlist.cql

• We have set our cassandra database with the data. We have to download JDBC .jar file (Download) and keep it in the $Neo4j_Home/plugins. We can provide URL in the $Neo4j_Home/conf/neo4j.conf as :

apoc.jdbc.cassandra_songs.url=jdbc:cassandra://localhost:9042/playlist

Restart the Neo4j server and we are set for migrating the data from the Cassandra to Neo4j.

• We hit the cassandra and start fetching data and perform count operation.

CALL apoc.load.jdbc('cassandra_songs','artists_by_first_letter') yield row
RETURN count(*);

• Let’s create Index, Constraints and Relation the data.

/**
* Here we define schema and key.
*/
CALL apoc.schema.assert(
  {Track:['title','length']},
  {Artist:['name'],Track:['id'],Genre:['name']});

• Now we will load data and perform Merge and Create operation so that we can create the node and relationship between the node.

/**
* Here we load data in the neo4j and create node with the help of schema which we define
* earlier.
*/
CALL apoc.load.jdbc('cassandra_songs','track_by_artist') yield row
MERGE (a:Artist {name:row.artist})
MERGE (g:Genre {name:row.genre})
CREATE (t:Track {id:toString(row.track_id), title:row.track,
length:row.track_length_in_seconds})
CREATE (a)-[:PERFORMED]->;(t)
CREATE (t)-[:GENRE]->(g);

• We can see Relation Graph and it will look something like this :

/**
* For Displaying Performed Relation
*/

MATCH p=()-[r:PERFORMED]->() RETURN p LIMIT 25;

/**
* For Displaying GENRE Relation
*/

MATCH p=()-[r:GENRE]->() RETURN p LIMIT 100;

After importing the data in Neo4j, we have to thing about the sync of data. We can use schedule process which can be timebase and automatically sync data between the databases. We can also used event based integration where we will defined the event at which we want to update the database.

Note : As we discuss I want to notify again if you do not update driver name into $Neo4j_Home/conf/neo4j.conf then you have to load driver in Neo4j otherwise you have to provide only driver name into the query.

This is an basic example for using the APOC and it is also a first step when you start using Neo4j and want to replace it with your old databases that time you don’t want to use loose your data. After migrating the data you are ready to use Neo4j with your data which was exists in the old databases.

If You have any questions you can contact me here or on Twitter: @anuragknoldus