The traditional application management system, that is, the interaction of applications with relational database using RDBMS, is one of the sources that generate Big Data.
Such Big Data, generated by RDBMS, is stored in Relational Database Servers in the relational database structure.
Sqoop − “SQL to Hadoop and Hadoop to SQL”
Sqoop is a tool designed to transfer data between Hadoop and relational database servers.
It is used to import data from relational databases such as MySQL, Oracle to Hadoop HDFS, and export from Hadoop file system to relational databases. It is provided by the Apache Software Foundation.
How Sqoop Works?
The following image describes the workflow of Sqoop.
Sqoop installation on Linux
Before installing Sqoop, make sure Hadoop is installed.
To install Sqoop follow the below steps:
Open the terminal and download the sqoop archive:
$ wget http://mirrors.estointernet.in/apache/sqoop/1.4.7/sqoop-1.4.7.tar.gz
This will download the Sqoop archive, Now extract the Sqoop files:
$ tar -xvf sqoop-1.4.7.tar.gz
Now you have to set Sqoop Environment Variables in .bashrc file.
$ export SQOOP_HOME=/home/edureka/sqoop-1.4.7 export PATH=$PATH:$SQOOP_HOME/bin
Now you have to configure Sqoop.
Follow the below steps to do this:
$ cd $SQOOP_HOME/conf
$ mv sqoop-env-template.sh sqoop-env.sh
Now open the sqoop-env.sh file with a text editor
$ nano sqoop-env.sh
and add the path to Hadoop:
The path might change depending on where you have stored Hadoop. Use the path applicable to you.
Save (Ctrl+o) and exit (Ctrl+x)
Next step is to configure the mysql connector.
Run the below commands to do it:
$ wget http://ftp.ntu.edu.tw/MySQL/Downloads/Connector-J/mysql-connector-java-8.0.15.tar.gz
$ tar -xvf mysql-connector-java-8.0.15.tar.gz
$ mv mysql-connector-java-8.0.15/mysql-connector-java-8.0.15.jar /$SQOOP_HOME/lib
Finally, to verify the installation, run:
$ sqoop -version
This will output version about sqoop
25/2/19 18:52:32 INFO sqoop.Sqoop: Running Sqoop version: 1.4.5
Sqoop 1.4.5 git commit id 5b34accaca7de251fc91161733f906af2eddbe83
Syntax
The following syntax is used to import data into HDFS.
$ sqoop import (generic-args) (import-args)
$ sqoop import (generic-args) (import-args)
Example
Let us take an example of three tables named as emp, emp_add, and emp_contact, which are in a database called userdb in a MySQL database server.
For More Information:https://www.tutorialspoint.com/sqoop/sqoop_introduction.html
