Transforming Data for Amazon Kinesis Firehose and S3 Bucket
Question
A company is planning on using Amazon Kinesis firehose to stream data into an S3 bucket.
They need the data to be transformed first before it can be sent to the S3 bucket.
Which of the following would be used for the transformation process?
Answers
Explanations
Click on the arrows to vote for the correct answer
A. B. C. D.Answer - B.
The AWS Documentation mentions the following.
Kinesis Data Firehose can invoke your Lambda function to transform incoming source data and deliver the transformed data to destinations.
You can enable the Kinesis Data Firehose data transformation when you create your delivery stream.
Option A is incorrect because SQS is a queue service and can not transform the data.
Option C is incorrect because although with EC2 instances, you can achieve the desired result, you need to manage and provision the server.
Lambda is a better option.
Option D is incorrect because an API gateway is used to make API calls and not transform it.
For more information on Kinesis Firehose, please refer to the below URL-
https://docs.aws.amazon.com/firehose/latest/dev/data-transformation.htmlThe correct answer is B. AWS Lambda.
AWS Kinesis Firehose is a fully managed service that can be used to capture, transform, and load streaming data in real-time into Amazon S3 or Amazon Redshift. Firehose can automatically convert the incoming data to a desired format before loading it into S3 or Redshift. This conversion is known as data transformation.
AWS Lambda is a serverless compute service that enables you to run code without provisioning or managing servers. You can use Lambda to run your custom code to transform the incoming data stream into a desired format before it is loaded into S3 using Kinesis Firehose.
Lambda can be triggered by Kinesis Firehose when new data is received. The Lambda function can then transform the data using custom code, such as JSON parsing or data normalization, before sending it to S3. Lambda can scale automatically to handle any amount of incoming data and can be used to apply custom transformations on data streams in real-time.
SQS is a message queuing service that allows you to decouple and scale microservices, distributed systems, and serverless applications. It is not designed for data transformation and is not a good fit for this scenario.
EC2 is a virtual server that can be used to run applications in the cloud. However, it requires manual provisioning and management of servers, which goes against the serverless approach of Kinesis Firehose and Lambda.
API Gateway is a fully managed service that makes it easy for developers to create, publish, maintain, monitor, and secure APIs at any scale. It is not designed for data transformation and is not a good fit for this scenario.
In conclusion, AWS Lambda is the best choice for transforming the data before it is loaded into S3 using Kinesis Firehose.
