In the present scenario, enterprises have data on the network, on the cloud, and on the endpoint. Thus enabling governance on data is a critical step to understand the sources governing data, last update on the data, classification of data, the relationship and linkage between data and data sources.
Apache Atlas helps in providing the ability to analyze the metadata and then take actions and apply appropriate policies as needed.
What is Apache Atlas?
“Atlas is a scalable and extensible set of core foundational governance services – enabling enterprises to effectively and efficiently meet their compliance requirements within Hadoop and allows integration with the whole enterprise data ecosystem. “
Apache Atlas is a data governance tool for data governance and metadata management on enterprise Hadoop clusters. It is one stop solution for gathering, processing and maintaining metadata.
Features
Centralized Metadata
Atlas provides the ability to define new metadata types and also facilitates easy exchange of metadata by enabling any metadata consumer to share a common metadata store.
Data Classification
Atlas provides the ability to dynamically create classifications- like PII, EXPIRES_ON, DATA_QUALITY, SENSITIVE. Classifications can include attributes – like expiry_date attribute in EXPIRES_ON classification
Lineage
It provides an Intuitive UI to view the lineage of data as it moves through various processes as well as a REST APIs to access and update lineage.
Terms and Components used in Apache Atlas
Types:
A ‘Type’ in Atlas is a definition of how a particular type of metadata objects are stored and accessed. It represents one or a collection of attributes that define the properties for the metadata object. Some already available types are:
- hive_table
- jms_topic
- avro_collection
- avro_schema
- storm_bolt etc.
Entities:
An ‘entity’ in Atlas is a specific value or instance of an Entity ‘type’ and thus represents a specific metadata object in the real world.
Type System:
It allows users to define and manage types and entities. All new types of metadata in Atlas are modeled using types and represented as entities.
Graph Engine:
Internally, Atlas persists metadata objects it manages using a Graph model. Graph engine component is responsible for translating between types and entities of the Atlas type system. Atlas uses the JanusGraph to store the metadata objects.
Ingest / Export:
The Ingest component allows metadata to be added to Atlas. Similarly, the Export component exposes metadata changes detected by Atlas to be raised as events. Consumers can consume these change events to react to metadata changes in real time.
Integration with Apache Atlas
Users can manage metadata in Atlas and create new Metadata models using these two methods:
Rest API
All functionality of Atlas is exposed to end users via a REST API that allows types and entities to be created, updated and deleted.
Messaging
In addition to the API, users can choose to integrate with Atlas using a messaging interface that is based on Kafka. Atlas uses Apache Kafka as a notification server for communication between hooks and downstream consumers of metadata notification events. Events are written by the hooks and Atlas to different Kafka topics.
Interacting with Atlas
To demonstrate the functionality of Apache Atlas, we will be using its REST API to create and read new entities.
Build and Install
To build and install Atlas, refer atlas installation steps.
Here, we will be using Apache Atlas package with embedded-hbase-solr profile, that includes Apache HBase and Apache Solr.
To run Apache Atlas with local Apache HBase & Apache Solr instances that are started/stopped along with Atlas start/stop, run following commands:
export MANAGE_LOCAL_HBASE=true export MANAGE_LOCAL_SOLR=true bin/atlas_start.py
Its UI is accessible on default port 21000, : http://localhost:21000
Create new Entity type in Apache Atlas via REST API
curl -u <username>:<password> -ik -H "Content-Type: application/json" -X POST -d '{"entity": {"typeName" : "hdfs_path", "attributes" : {"name" : "electionresults", "qualifiedName" : "/user/dmaster/electionresults/ls2014.tsv@Sandbox", "path" : "/user/dmaster/electionresults/ls2014.tsv", "clusterName":"Sandbox"}}}' http://localhost:21000/api/atlas/v2/entity
List all the entities in a given type
curl -u admin:admin -X GET http://localhost:21000/api/atlas/entities?type=hdfs_path
{"requestId":"pool-1-thread-8 - e4f2d5ad-91dc-4776-a1c5-94b81b7e6859","typeName":"hdfs_path","count":1,"results":["b1ed9bc1-e6e7-48f6-ac70-578ecb13ce79"]}
Delete an entity from Atlas using its GUID
curl -iv -u admin:admin -X DELETE http://localhost:21000/api/atlas/entities?guid=b1ed9bc1-e6e7-48f6-ac70-578ecb13ce79
In the next blog, we will talk about Apache Atlas Client and utilizing it for metadata management and governance.
