Your company, a financial services firm, has asked your team to build an analytics and machine learning platform to analyze and forecast your company's trading operations using Athena, S3, and SageMaker Studio.
The volume of data received on a daily basis is very high.
The data, stored in S3, will be used as feature data for your machine learning model that uses the XGBoost SageMaker built-in algorithm.
The source systems that stream data into your environment send their data in JSON format in real-time.
Your team needs to transform the data in real-time to prepare it for your machine learning model.
Before storing it on S3 for use in your SageMaker XGBoost algorithm-based model, how can you transform the data to prepare it for training?
Click on the arrows to vote for the correct answer
A. B. C. D.Correct Answer: A.
Option A is correct.
This option satisfies the real-time requirement while also being the most efficient and requiring the least amount of effort for your team.
Also, the XGBoost algorithm only supports the libsvm and CSV content types for training and inference.
Option B is incorrect.
This option can meet your real-time requirement, but it is far more complex to set up and maintain for your team than using the Kinesis Data Streams and Kinesis Data Firehose option.
Also, the XGBoost algorithm only supports the libsvm and CSV content types, not the x-recordio-protobuf content type for training and inference.
Option C is incorrect.
This option is incorrect because Glue ETL jobs imply batch processing, which fails to meet your real-time requirement.
Also, the XGBoost algorithm only supports the libsvm and CSV content types, not the x-recordio content type for training and inference.
Option D is incorrect.
This option is also incorrect because it is far more complex to set up and maintain for your team than using the Kinesis Data Streams and Kinesis Data Firehose option.
Also, the XGBoost algorithm only supports the libsvm and CSV content types, not the x-recordio content type for training and inference.
References:
Please see the AWS blog titled Archiving Amazon MSK Data to Amazon S3 with the Lenses.io S3 Kafka Connect Connector (https://aws.amazon.com/blogs/apn/archiving-amazon-msk-data-to-amazon-s3-with-the-lenses-io-s3-kafka-connect-connector/),
The Amazon SageMaker developer guide titled Prepare ML Data with Amazon SageMaker Data Wrangler (https://docs.aws.amazon.com/sagemaker/latest/dg/data-wrangler.html),
The Amazon Kinesis Data Firehose developer guide titled Converting Your Input Record Format in Kinesis Data Firehose (https://docs.aws.amazon.com/firehose/latest/dev/record-format-conversion.html),
The Amazon SageMaker developer guide titled Common Data Formats for Training (https://docs.aws.amazon.com/sagemaker/latest/dg/cdf-training.html),
The Amazon SageMaker developer guide titled XGBoost Algorithm (https://docs.aws.amazon.com/sagemaker/latest/dg/xgboost.html)
The task at hand is to transform real-time JSON data received from source systems into a format suitable for training a machine learning model using SageMaker XGBoost algorithm. The transformed data will be stored in S3 for use in the machine learning model. The volume of data received daily is very high, which means the transformation process must be efficient and scalable.
Option A suggests using Kinesis Data Streams to ingest the JSON data from the source systems. Kinesis Data Firehose can then be used to receive the data from Kinesis Data Streams. A Lambda function can be leveraged to convert the JSON data into libsvm format. Then, a Kinesis Data Firehose transform can be used to write the data to S3.
Kinesis Data Streams is a scalable and durable real-time data streaming service that can collect and process large amounts of data in real-time. Kinesis Data Firehose is a fully managed service that can load streaming data into S3 or other destinations, such as Redshift or Elasticsearch. Lambda functions are serverless functions that can be used to process data in real-time. The libsvm format is a widely used sparse data format for machine learning algorithms.
Option B suggests using Apache Spark Structured Streaming in an EMR cluster to ingest the JSON data from the source systems. Apache Spark can then be used to convert the JSON data into x-recordio-protobuf format.
Apache Spark is a powerful distributed computing framework for processing large-scale data. EMR is a fully managed service that can be used to provision and scale clusters running Apache Spark and other big data frameworks. Structured Streaming is a scalable and fault-tolerant stream processing engine built on top of Apache Spark. x-recordio-protobuf is a binary data format that can be used to serialize data for use in machine learning models.
Option C suggests using Kinesis Data Streams to ingest the JSON data from the source systems. Glue ETL job can then be used to convert the data from JSON into x-recordio format.
Glue is a fully managed ETL service that can be used to extract, transform, and load data from various sources into data lakes, data warehouses, and other destinations. It can automatically generate ETL code in Python or Scala that can be executed on Spark or other big data engines. x-recordio is a binary data format that can be used to serialize data for use in machine learning models.
Option D suggests using Apache Kafka Streams running on EC2 instances to ingest the JSON data from the source systems. The Kafka Connect S3 connector can then be used to serialize the data onto S3 as x-recordio.
Kafka is a distributed streaming platform that can be used to collect and process real-time data streams. Kafka Streams is a library built on top of Kafka that can be used to build scalable and fault-tolerant stream processing applications. EC2 is a scalable and secure compute service that can be used to provision virtual servers in the cloud. Kafka Connect is a framework for integrating Kafka with external systems. The S3 connector is a Kafka Connect plugin that can be used to write data from Kafka topics to S3. x-recordio is a binary data format that can be used to serialize data for use in machine learning models.
Overall, all the options offer viable ways to transform the JSON data into a suitable format for training the machine learning model. The choice of which option to use depends on the specific requirements and constraints of the project, such as the volume of data, the desired latency, the available resources, and the skill set of the team.