HIve MetaStore Event Listeners are used to Detect the every single event that takes place whenever an event is executed in hive, in case You want some action to take place for an event you can override MetaStorePreEventListener and provide it your own Implementation

in this article, we will learn how to create our own metastore event listeners in the hive using scala and sbt

so let’s get started first add the following dependencies in your build.sbt file

libraryDependencies += "org.apache.hive" % "hive-exec" % "1.2.1" excludeAll
  ExclusionRule(organization = "org.pentaho")

libraryDependencies += "org.apache.hadoop" % "hadoop-common" % "2.7.3"

libraryDependencies += "org.apache.httpcomponents" % "httpclient" % "4.3.4"

libraryDependencies += "org.apache.hadoop" % "hadoop-client" % "2.6.0"

libraryDependencies += "org.apache.hive" % "hive-service" % "1.2.1"

unmanagedJars in Compile += file("/usr/lib/hive/lib/hive-exec-1.2.1.jar")

assemblyMergeStrategy in assembly := {
  case PathList("META-INF", xs @ _*) => MergeStrategy.discard
  case x => MergeStrategy.first
}

now create your first class you can be named it anything I named it as OrcMetastoreListener this class must extend  MetaStorePreEventListener class of hive and take Hadoop conf as the constructor argument

import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.hive.metastore.MetaStorePreEventListener
import org.apache.hadoop.hive.metastore.events.PreEventContext.PreEventType._
import org.apache.hadoop.hive.metastore.events._

class OrcMetastoreListener(conf: Configuration) extends MetaStorePreEventListener(conf) {

  override def onEvent(preEventContext: PreEventContext): Unit = {
    preEventContext.getEventType match {
      case CREATE_TABLE =>
        val tableName = preEventContext.asInstanceOf[PreCreateTableEvent].getTable
        tableName.getSd.setInputFormat("org.apache.hadoop.hive.ql.io.orc.OrcInputFormat")
        tableName.getSd.setOutputFormat("org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat")
      case ALTER_TABLE =>
        val newTableName = preEventContext.asInstanceOf[PreAlterTableEvent].getNewTable
        newTableName.getSd.setInputFormat("org.apache.hadoop.hive.ql.io.orc.OrcInputFormat")
        newTableName.getSd.setOutputFormat("org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat")
      case _ => //do nothing

    }

  }
}

the pre-event context contains all the hive meta store event in my case I want that whenever a table is get created in the hive it must use hive input format and output format and same thing for altering command

The best use case for this listener is when somebody wants to query a data source such as spark or any other data source using its own custom input format and even don’t want to alter the schema  of hive table to use his custom input format

now let’s build a jar from the core and use it in the Hive

First, add sbt assembly plugin in your plugins.sbt file

addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.5")

now got to your root project and build the jar with command sbt assembly

it will build your jar, collect your jar and put it in your $HIVE_HOME/lib path

inside $HIVE_HOME/conf folder add the following contents in hive-site.xml
<configuration>
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:derby:metastore_db;create=true</value>
<description>JDBC connect string for a JDBC metastore</description>
</property>
<property>
<name>hive.metastore.schema.verification</name>
<value>false></value>
</property>
<property>
<name>hive.metastore.pre.event.listeners</name>
<value>metastorelisteners.OrcMetastoreListener</value>
</property>
</configuration>

now create a table in hive and describe it

hive> CREATE TABLE HIVETABLE(ID INT);
OK
Time taken: 2.742 seconds
hive> DESC EXTENDED HIVETABLE
    > ;
OK
id                  	int                 	                    
	 	 
Detailed Table Information Table(tableName:hivetable, dbName:default, owner:hduser,e, inputFormat:org.apache.hadoop.hive.ql.io.orc.OrcInputFormat, outputFormat:org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat, compressed:false, 
Time taken: 0.611 seconds, Fetched: 3 row(s)

I expect you to be familiar with basics of Jenkins. If you’re not please visit Introduction to Jenkins, this post will take you through very basics of Jenkins. What I want to introduce to you, are post setup type of things. Means, you have already setup Jenkins and now you are worried about how to do pipelining of the project.

I will take you through the problems that might come when you pick working on a Jenkins Pipeline project that uses Docker images. To understand what is a Docker Image please visit Introduction to Docker, this post will take you through the basics of Docker Image.

Pipe-Lining a Maven Project:

Creating a Pipe-line is similar to create a simple Jenkins Job, but in here you have to give some different configurations for your job. Let’s start.

Steps:

1. Goto Jenkins Home and click new Item.

2. Select the pipeline option and give a suitable job name and press OK.

3. Now give proper configurations for this Job as defined below:

a. In General tab you can give the project based security to a particaular person/group of people and define what role/permissions you want this person/group to be involved in.

b. You don’t need to touch other settings in General/Job Notification/Office 365 Connector Tabs for a simple pipeline.

c. In next Tab i.e. Build Triggers, you can define the type of trigger you want to automate to build your Job or you can leave it blank if you want to manually trigger your Build.

d. Then the most important configuraion is Pipeline tab.

i) There are two ways to make pipeline, first you can write a script in given textbox in Pipeline Script option, second you can select Pipeline Script from SCM, and provide a Jenkinsfile in your project and give the path in script path.

ii) Then define your Source Code Management(SCM) in SCM option i.e. Git in our case.

iii) Then define your Git Repository URL, your will see an error as show below in the image steps, the next image show how to resolve it. You’ll have to create a proper Jenkins credential for the given Repo and select it in Credential option. The error will then disapear.

Continue reading →

Hi all,

Knoldus has organized a 30 min session on 18th August 2017 at 3:30 PM. The topic was “An Introduction to Jenkins”. Many people have joined and enjoyed the session. I am going to share the slides and the video of the session here. Please let me know if you have any questions related to linked slides or doubts regarding the content.
Here are the slides:

And Here’s the video of the session:

In case you have any doubts regarding the topic you may ask them in the comment section below.