Migrating Hadoop Environment to the Cloud
Question
Your company is planning to migrate their on-premises Hadoop environment to the cloud.
Increasing storage cost and maintenance of data stored in HDFS is a major concern for your company.
You also want to make minimal changes to existing data analytics jobs and existing architecture.
How should you proceed with the migration?
Answers
Explanations
Click on the arrows to vote for the correct answer
A. B. C. D.D.
The best option for migrating an on-premises Hadoop environment to the cloud, while minimizing changes to existing data analytics jobs and architecture and addressing concerns about increasing storage costs and data maintenance, is to create a Cloud Dataproc cluster on Google Cloud Platform and migrate the Hadoop environment to the new cluster, and move the data to Cloud Storage.
Option A: Migrate data stored in Hadoop to BigQuery
Migrating data stored in Hadoop to BigQuery would require significant changes to existing data analytics jobs and architecture, as the structure and access patterns of data in BigQuery are different from those in Hadoop. Additionally, BigQuery is a fully-managed service, so you would not have the same level of control over your data as you would in a Hadoop environment.
Option B: Create Compute Engine instances with HDD instead of SSD
Creating Compute Engine instances with HDD instead of SSD would save on storage costs, but it would not address the issue of data maintenance and would require a full migration of the existing environment into the new one in Compute Engine instances. This would also require significant changes to existing data analytics jobs and architecture, as the structure and access patterns of data in Compute Engine instances are different from those in Hadoop.
Option C: Create a Cloud Dataproc cluster on Google Cloud Platform and move HDFS data into larger HDD disks
Creating a Cloud Dataproc cluster on Google Cloud Platform and moving HDFS data into larger HDD disks would address the issue of storage costs, but it would not address the issue of data maintenance. Additionally, it would require significant changes to existing data analytics jobs and architecture, as the structure and access patterns of data in Cloud Dataproc are different from those in Hadoop.
Option D: Create a Cloud Dataproc cluster on Google Cloud Platform, migrate Hadoop code objects to the new cluster, and move data to Cloud Storage
Creating a Cloud Dataproc cluster on Google Cloud Platform, migrating Hadoop code objects to the new cluster, and moving data to Cloud Storage is the best option for migrating an on-premises Hadoop environment to the cloud while minimizing changes to existing data analytics jobs and architecture and addressing concerns about increasing storage costs and data maintenance. This option allows for Hadoop code objects to be migrated to the new cluster, minimizing changes to existing data analytics jobs and architecture. Moving data to Cloud Storage addresses the issue of increasing storage costs and data maintenance, as Cloud Storage is a highly durable and scalable object storage service. Leveraging the Cloud Dataproc connector to run jobs on that data allows for easy integration with the new environment.
