EMR Hadoop Ecosystem
Question
Allianz Financial Services (AFS) is a banking group offering end-to-end banking and financial solutions in South East Asia through its consumer banking, business banking, Islamic banking, investment finance and stock broking businesses as well as unit trust and asset administration, having served the financial community over the past five decades. AFS launched EMR cluster to support their big data analytics requirements.
AFS is looking at a metadata management tool that allows you to access Hive metastore tables within Pig, Spark SQL, and/or custom MapReduce applications.
The component is similar to Glue Data Catalog. Which EMR Hadoop ecosystem fulfills the requirements?
Answers
Explanations
Click on the arrows to vote for the correct answer
A. B. C. D.Answer : C.
Option A is incorrect.
Hive is an open-source, data warehouse, and analytic package that runs on top of a Hadoop cluster.
Hive scripts use an SQL-like language called Hive QL (query language) that abstracts programming models and supports typical data warehouse interactions.
Hive enables you to avoid the complexities of writing Tez jobs based on directed acyclic graphs (DAGs) or MapReduce programs in a lower level computer language, such as Java.
Hive extends the SQL paradigm by including serialization formats.
You can also customize query processing by creating table schema that matches your data, without touching the data itself.
In contrast to SQL (which only supports primitive value types such as dates, numbers, and strings), values in Hive tables are structured elements, such as JSON objects, any user-defined data type, or any function written in Java.
https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-hive.htmlOption B is incorrect.
HBase is an open source, non-relational, distributed database developed as part of the Apache Software Foundation's Hadoop project.
HBase runs on top of Hadoop Distributed File System (HDFS) to provide non-relational database capabilities for the Hadoop ecosystem.
HBase works seamlessly with Hadoop, sharing its file system and serving as a direct input and output to the MapReduce framework and execution engine.
HBase also integrates with Apache Hive, enabling SQL-like queries over HBase tables, joins with Hive-based tables, and support for Java Database Connectivity (JDBC).
https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-hbase.htmlOption C is correct.
HCatalog is a tool that allows you to access Hive metastore tables within Pig, Spark SQL, and/or custom MapReduce applications.
HCatalog has a REST interface and command line client that allows you to create tables or do other operations.
https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-hcatalog.htmlOption D is incorrect.
Apache Phoenix is used for OLTP and operational analytics, allowing you to use standard SQL queries and JDBC APIs to work with an Apache HBase backing store.
https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-phoenix.htmlThe component mentioned in the question is a metadata management tool that provides access to Hive metastore tables within Pig, Spark SQL, and custom MapReduce applications. This indicates that the solution should be able to integrate with Hive metastore and provide metadata management capabilities to other tools.
Apache Hive is a data warehousing framework that enables SQL-like querying of data stored in Hadoop. It uses a metastore to store schema and metadata information about tables and partitions. Therefore, Hive can be used to access Hive metastore tables within Pig, Spark SQL, and custom MapReduce applications. Hive also provides a command-line interface to manage metadata and schema information.
Apache HBase is a distributed, column-oriented database that provides random real-time access to Big Data. It is not a metadata management tool and does not integrate with Hive metastore.
Apache HCatalog is a table and storage management layer for Hadoop that enables users to share data across multiple Hadoop applications. It provides a centralized metadata management system and allows integration with Hive. HCatalog can be used to access Hive metastore tables within Pig and MapReduce applications but does not support Spark SQL.
Apache Phoenix is an open-source, relational database layer on top of HBase that provides low-latency SQL queries for Hadoop. It does not provide metadata management capabilities and does not integrate with Hive metastore.
Therefore, the correct answer to the question is C. Apache HCatalog, as it provides a metadata management tool that allows access to Hive metastore tables within Pig, Spark SQL, and custom MapReduce applications, similar to Glue Data Catalog.
